A System for Interactive Multimedia Retrieval Evaluations

The evaluation of the performance interactive multimedia retrieval systems is a methodologically non-trivial endeavour and requires specialized infrastructure. Current campaigns have so far relied on local setting, where all needed to be evaluated at same physical location time. This constraint does not only complicate organization coordination but also limits number which can reasonably within set time frame. Travel restrictions might further limit possibility for such evaluations. To address these problems, evaluations need conducted in (geographically) distributed was possible due lack supporting In this paper, we present Distributed Retrieval Evaluation Server (DRES), an open-source system facilitate both traditional on-site as well fully settings has already proven effective competitive evaluation.